Category: Noteable

Service Feedback, how to ‘git gud’

Running a service like Noteable comes with its own particular set of problems. Academic colleagues who have used the service in their teaching always say that the service is great, which itself is very good, the problem comes when trying to solidify this into something that you can take to others.

I’ve realised that it sounds like I’m complaining that people like our service, let me clarify, the problem is getting other people to believe me and get on board as well. Within academia it can be difficult to find time to experiment with something new, one of the best parts of being a Learning Technologist is to make it easier for colleagues to find tools and services that suit their needs and help them to deliver teaching.

This is how I meander my way to talking about feedback, the process of documenting how people found using your service; turning this into things that you can work on and then using it to (identify and then) shout about what your service is good at.

So, you’ve decided to do some service feedback. Here’s what you should know…

I spent a lot of time trying to find some sort of template for what questions to ask but I ran up a blank, I also found a collection of incredibly strange/weird questions which I had to fight the urge to include purely to indulge myself: “If our service were a car, what brand of car would it be?”

Since I couldn’t find anything I decided to try and make my own and then share the template out for others to improve on. I’ll break this down into two components, questionnaire feedback and User Stories/Use Cases (call them what you want but in the end it’s a first-hand account of what your users do with the service and how they found it). Today I’m going to focus on the first.

Questionnaire Feedback:

Questionnaires are usually a bit of a go to, they don’t take as long and you can normally get so numbers which look good on a page at the end of it. They (can be) a quick way of finding things out and the information can be useful, if you have the right questions that is…

In thinking about what the right questions are you really need to think about; what information do you want to know and do these questions allow your users to give you that information. For the vast majority of you reading this that will be very obvious but it still bears thinking about. Below, i’ll list all of the questions that I eventually created alongside the rationale.

How have you used the Noteable service in your teaching? (Free text answer)

A basic opener to get a bit more information about how your users have been engaging with your service. This sounds like a fairly bland opener but it does a number of things; gives you context information and also gives you a handy list of “this service can be used for…”

Do you intend to use the Noteable service again? (Free text answer)

Very straight to the point, this service was offered in a trial phase so it’s important to know how many people thought it was good enough to want to use again. This gives you direct retention numbers and helps you to justify that you are fulfilling a continuing need.

How likely are you to recommend the Noteable service to a colleague who has similar teaching needs? (Free text answer)

For a service that wants to grow this is key. People might like your service but if they really like it then they will suggest it to their colleagues. It’s a good metric as it’s a great thing to brag about, essentially it’s not just you that is saying that the service is great but also the users. Users will always be more open to hearing how good a service is from their colleagues so there is also the implication that the service will continue to grow naturally.

How would you rate the ease of use of the Noteable service? (Scale Question 1Low-5High)

Ease of use means how easily is it for you to use each time you use it in your teaching. You want to know if your users constantly feel like they are straining to use your service. Making the question more open like this means you aren’t highlighting what you think are barriers but looking out for anything that could be. A caveat with these questions is that any score under 3 will open a follow up question asking about specifics.

How would you rate the ease of adoption? (Scale Question 1Low-5High)

This is in relation to how much effort it took to introduce your service into their teaching. You obviously want to pay attention to this as it is going to impact the amount of new users and the early attrition rate. Again, this has a follow up if it gets a low mark to get more useful information on what you can improve on. Removing or lowering as many barriers to adoption is going to be the biggest thing to help you grow a service.

What would you do without the Noteable service? (Free text answer)

This might be one that is more suited to our position with the Noteable service. We are offering a centralised service so it’s good to know if this is reducing a burden or making it easier for people to use this specific tool – Jupyter notebooks. As a pilot service we need to be able to say there is a justifiable need for providing this service, that without this service users would be inconvenienced.

What is the most important feature that you would like to see? (Free text answer)

Nothing like crowd-sourcing your new features….

To what extent do you agree with the following statement: “Using Noteable helped me to deliver my teaching/training” (Scale Question 1Low-5High)

Directly asking if introducing this tool has any discernible affect on the ability of our colleagues to actually deliver their teaching. It’s often very difficult to quantify where something has a pedagogic impact but by asking this directly we are getting the answer from the people who are most likely to know.

To what extent do you agree with the following statement: “Using Noteable helped my students to interact with the material” (Scale Question 1Low-5High)

Again, this is something that is difficult to quantify but can have a massive impact. If your students can more easily engage with the material it means you can get further into more complex subject, spend more time in theory and less time getting over barriers. This will be followed up with student questionnaires at a later date.

It’s important to not lose sight of how valuable this information can be, whilst you will want to shout about good feedback make sure you spend the time fixing what users have reported and working on what they have said they want in the future. Few things can improve your relationship with your users more than “you said, we did”.

This questionnaire is currently live, I will share the feedback when it is finally complete. Please let me know if you have any improvements or insights into this, I’m always trying to make this better.

p.s. if you are going to steal these questions, at least replace Noteable with [your service name]. (I have actually seen this before and it is amazing/terrible)


More about Noteable

I’ve recently tried to become a little more involved in the wider Jupyter community and I have to say that it’s the most amazing community to be part of and work with. We at the University of Edinburgh have already managed to get funding for our nbgrader workshop which has been a great excuse to introduce the University, myself and the Noteable service.

If you want to get involved, or even just be in on the discussions then first head over to Jupyter Discourse, secondly make sure to get in on the newly revived Jupyter Community meetings. The community is always full of interesting nuggets or information and with such a rapidly growing community there is always something new coming up. If you worry about not being able to keep up then I can highly advise the Tracking Jupyter Newsletter by Tony Hirst

Being involved in the community has already started to pay off with the following discovery, originally found on Discourse –

This is a great intro to what’s out there is the Jupyter ecosystem, a breakdown comparison of 6 services that allow you to run/share/preview Jupyter notebooks……but Oh Dear, Noteable isn’t on the list! Time to get our name up there with the big companies like Microsoft and Google. So here is my contribution to how Noteable compares, including some bonus points! Noteable is a service designed for use in educational institutions, currently looking for partners in the UK.

Supported languages: Python 2.7, Python 3.5, Julia, R, Sage, Matlab, Calysto, Octave, Haskell (also able to install other languages by request!)
Ability to install packages: Yes, for the duration of your session. Also we can install packages by request to avoid having to install them yourself (also we have a lot already –
Interface similarity: Uses Native Jupyter notebook interface
Keyboard shortcuts: Uses all of the same keyboard shortcuts as Jupyter
Missing features: None
Added features: nbgrader (currently in Beta)built using Docker containers to allow you to switch between environments quickly by launching a new container.
Ease of working with datasets: You can upload a dataset to your project from your local computer, and it can be accessed by any notebook.
Internet access: Yes
Ability to work privately: Yes
Ability to share publicly: Not directly but you can use Git to share your repos or send notebooks manually.
Ability to collaborate: Not currently
Ability to upgrade for better performance: As a service designed for education there is no ‘free’ version but we do offer trial access to Noteable to institutions that are interested.
Documentation and technical support: General documentation for integrating with your LMS via LTI is available. General introduction documentation to using Jupyter is currently being created.
Conclusion: Noteable is the only service developed and designed for use in education, primarily further and higher education. We have already developed a working LTI connection to fit in with your LMS and work closely with our users to add features that are of the most benefit.

If what you see above sounds like something you would be interested in then feel free to get in touch and ask for a trial for your institution.


After following the original thread on Discourse and the Google Group there have been some extra questions raised:

– Jupyter Authenticator Support: Can it work with my organizations’ existing single sign on app? e g. 2FA, TOTP, the new Webauthn, GitHub, GitLab

Noteable can work with your institutions login, with a working LTI we can also directly connect with your LMS meaning that your users can be seamlessly logged in and also directed to specific course (which becomes important with our new Assignments)

– Mobile interface: Can any of the existing iOS and Android front ends connect to the system?

Directly from Noteable we do not offer our own App but I can say that we work on Juno, this is a paid App but does allow you to try some basic notebooks for free.  They have also recently announced that they will be donating back a portion of their profits to Jupyter so a win all around.

– Backups: How does the system backup work? How frequently, to where, and are there backups tested to make sure that they work?

Noteable auto-saves your work at regular intervals (every few minutes if you are making changes). You can also choose to save a Checkpoint if you think you might need to revert.

And we’re off!

The Noteable service is into its next big pilot phase this Semester. Over 500 students will be using the service this week across 6 different courses in 6 different School within the University of Edinburgh.

But what is the Noteable service and what is the scope of this pilot? I’m glad you asked (You did ask, I heard you)

Here’s a quick summary of what the pilot involves, what we will be hoping to achieve and how we will measure it. Comments are more than welcome. (Nice ones)


What will be delivered

The Noteable service is a cloud-based application providing access to Jupyter notebooks online. Noteable provides a central space to store and run Jupyter notebooks in a variety of languages.

The purpose of Noteable is to allow students and staff to access Jupyter notebooks at any time without the need for pre-installation which can be cumbersome and difficult for programming novices. Noteable is integrated with the institutional VLE to allow for a central launch point into a pre-set environment without the need for a separate login.


The initial pilot will run for the duration of Semester 1 of the 2018-19 Academic Year concluding in December 2018. Feedback information from staff and students as well as usage data and indications of further demand will be collected and presented before the end of December 2018.

Users – Students, Courses, Schools

For the pilot of this service, we have worked to ensure pilot users from across the University. The Semester 1 pilot will involve 580 students in 6 courses from 6 different School within the University.

We are also aware that the Noteable service will be used as part of the Digital Skills Programme and we will report of the number of students involved in these sessions at the end of the semester.


There are multiple benefits to be derived from the Noteable service pilot. Firstly, staff who already incorporate Jupyter notebooks into their teaching will benefit from removing the need to have students install Jupyter beforehand which is time-consuming and can cause issues, especially when using multiple additional packages. We will work alongside current users in the creation of Jupyter specific OER materials which will help new users be more easily able to adopt the service.


Success Criteria

The key success criteria for the Noteable service pilot will be based around 3 components:

  1. Demonstrate Need
  2. Service Fit
  3. Service Cost
 1. Demonstrate Need

The first goal of the Noteable pilot will be to determine that there is a need for a centrally supported notebook service for teaching purposes. This can be quantified both with the number of current users as well as secured future users. This will be broken down into two measurable indicators: number of courses using Noteable and number of students within these courses as a total. There will also be an additional measure citing the number of different Schools associated with the pilot to help ensure that the service is widely accepted across the University.

Goal: 6 Courses with up to 500 students in Semester 1

 2. Service Fit

Define whether the Noteable service fits the needs of the user community. This includes comparing the Noteable service with other comparable services. This criterion cannot be easily quantified and will largely be based on the feedback from the current and prospective user community. As part of this evaluation, there will be a suggestion as to whether to use Noteable or use a comparable service.

Goal: Create a comparison document with service recommendation

 3. Service Cost

Determine the total cost of running the Noteable service including the cost of staff (both development and support) and operational/infrastructure cost. This can be determined as a yearly cost and also broken down to a per-user cost. This defined cost can then be used as part of the comparison with other comparable services as per criterion 2.

Goals: Cost per year and Cost per user.


And to makes all this seem a little more real, here’s a graph of the use for Tuesday

Jupyter notebook Usage Graph
First ‘big’ usage day, peaking at over 100 concurrent Jupyter notebooks. The best bit is; this isn’t even our busiest day.

ProgTeach Play by Play

The first ProgTeach Symposium took place on the 24th of August, this has been something that colleagues and I have been working towards for some time so it was great to finally see this come together. The premise is fairly simple, across the University of Edinburgh lots of people teach computation in some form so let’s make time for getting as many of these people in the same room together and discuss what they do (and how they do it).

As my first main event that I’ve run at the University of Edinburgh, my main concern was that I had put something together that no one would want to attend…..we filled 80% in 24 hours. As you can see below the event was pretty well attended, what you can’t see is how diverse the audience was, colleagues from 13 different schools turned up with a smattering of units within those schools.

So to kick us off we had an introduction from Anne-Marie Scott, Deputy Director of Learning, Teaching and Web Services. Anne-Marie has been very keen on the idea of running these types of events to get people talking together about their teaching practice and is also my boss’s boss so no pressure.

Up next our first speaker was Dr Areti Mantaki from the Centre for Medical Informatics to talk about her experience of delivering large (huge) scale distance learning introduction to programming courses.

Areti shared some great incites about teaching coding generally and also some of the challenges (and surprises) of teaching entirely online. Something that a lot of attendees agreed with was how important it was to include mistakes in your examples but also spend time walking through the solution.

Next up we had Professor Chris Sangwin and Dr Stuart King, both from the School of Mathematics who talked about their introductory programming course. This including speaking about introducing Jupyter notebooks into their teaching as well as using CodeRunner as an assessment tool. Important to note that the students really enjoyed the ‘game’ of pursuing getting full marks in practice question sets.

Then it was my turn (James Slack) to talk about the University’s Noteable service currently in a pilot phase. For those not familiar with Noteable this is the University’s own JupyterHub environment, giving access to Jupyter notebooks for teaching purposes. Although this is still in a pilot phase the demand has been high and we will be delivering the service to over 500 students this coming semester.

The next speaker was Andrew Kirk from the Digital Skills and Training Team who are using the Noteable service as part of their Digital Skills Programme to deliver beginner Python courses to students from across the University. The courses were overwhelmingly popular, the first session had 90 bookings for only 20 places and will run again in the coming semester.

After the lunch break, we had a quick talk from Sean McGeever about the carpentries workshops in Edinburgh and the creation of a supportive RSE community. The theme of this talk about building a community to support colleagues from across the University was very closely aligned with the whole theme of the day. Sean also got the funding for cake so bonus points for Sean.

For the penultimate session, we wanted to focus on the on the discussion between colleagues about how they approach certain challenges of teaching computational content. Ironically no-one had the confidence the attempts to answer “How do you build confidence in learners”.

Heres a full breakdown of the discussion points – Discussion (Word Document)

And then to wrap up the day we had a quick talk about creating Jupyter based OERs to support new users, there is obviously a lot of material out there for ‘Intro to Python’ already but what can be really useful is a collection of these in notebook form that we know will work straight away in Noteable and also some ancillary materials about how to use notebooks to make it easier for people to introduce them into their teaching.

And that was that! A very interesting line up of talks and very useful discussion between disciplines which is something I’m always interested in promoting. I’ll leave it to Anne-Marie to wrap up the day:

And yes, there will be another one!


















Service Comparisons

A large part of my current project involves looking at other companies or services that offer Jupyter notebooks. The University has decided that it wants to provide notebooks but we also have to justify creating our own service. If there is something else that does what we need is it worth making our own version of it?

With this in mind, I have been creating a shortlist of services that are comparable and also drawing up a list of criteria to compare.

Here is the list of services:

Microsoft Azure Notebooks

Google Colaboratory


Anaconda Enterprise

And here’s a list of the criteria against which they will be compared:

Pricing  (how much do they charge and how do they charge it, what do you get access to when you pay, is there a free tier etc.)

Technical Specifications (e.g. RAM, CPU cores, Storage)

Extensions Available (e.g. nbgrader, RISE)

Supported Languages (There are many varied use cases throughout the university so how do we make sure all are supported, how quickly does the service add requested kernals.)

Focus/Audience (e.g. Azure is focused on Machine Learning, Goolge Colab is focused on collaborative programming)

Business Model (e.g. Azure is currently in a preview stage and may not be continued)

Community (This includes using the community as a measure of sector involvement and also how quickly the service administrators respond to feature requests or issues)

This comparison will take place over the summer and then I will publish our results. This is all about making sure that we are providing the best service that suits the needs of our users.

If you have anything you would like to add or think we have left any service or criteria out then let me know.