Month: March 2019

Service Feedback, how to ‘git gud’

Running a service like Noteable comes with its own particular set of problems. Academic colleagues who have used the service in their teaching always say that the service is great, which itself is very good, the problem comes when trying to solidify this into something that you can take to others.

I’ve realised that it sounds like I’m complaining that people like our service, let me clarify, the problem is getting other people to believe me and get on board as well. Within academia it can be difficult to find time to experiment with something new, one of the best parts of being a Learning Technologist is to make it easier for colleagues to find tools and services that suit their needs and help them to deliver teaching.

This is how I meander my way to talking about feedback, the process of documenting how people found using your service; turning this into things that you can work on and then using it to (identify and then) shout about what your service is good at.

So, you’ve decided to do some service feedback. Here’s what you should know…

I spent a lot of time trying to find some sort of template for what questions to ask but I ran up a blank, I also found a collection of incredibly strange/weird questions which I had to fight the urge to include purely to indulge myself: “If our service were a car, what brand of car would it be?”

Since I couldn’t find anything I decided to try and make my own and then share the template out for others to improve on. I’ll break this down into two components, questionnaire feedback and User Stories/Use Cases (call them what you want but in the end it’s a first-hand account of what your users do with the service and how they found it). Today I’m going to focus on the first.

Questionnaire Feedback:

Questionnaires are usually a bit of a go to, they don’t take as long and you can normally get so numbers which look good on a page at the end of it. They (can be) a quick way of finding things out and the information can be useful, if you have the right questions that is…

In thinking about what the right questions are you really need to think about; what information do you want to know and do these questions allow your users to give you that information. For the vast majority of you reading this that will be very obvious but it still bears thinking about. Below, i’ll list all of the questions that I eventually created alongside the rationale.

How have you used the Noteable service in your teaching? (Free text answer)

A basic opener to get a bit more information about how your users have been engaging with your service. This sounds like a fairly bland opener but it does a number of things; gives you context information and also gives you a handy list of “this service can be used for…”

Do you intend to use the Noteable service again? (Free text answer)

Very straight to the point, this service was offered in a trial phase so it’s important to know how many people thought it was good enough to want to use again. This gives you direct retention numbers and helps you to justify that you are fulfilling a continuing need.

How likely are you to recommend the Noteable service to a colleague who has similar teaching needs? (Free text answer)

For a service that wants to grow this is key. People might like your service but if they really like it then they will suggest it to their colleagues. It’s a good metric as it’s a great thing to brag about, essentially it’s not just you that is saying that the service is great but also the users. Users will always be more open to hearing how good a service is from their colleagues so there is also the implication that the service will continue to grow naturally.

How would you rate the ease of use of the Noteable service? (Scale Question 1Low-5High)

Ease of use means how easily is it for you to use each time you use it in your teaching. You want to know if your users constantly feel like they are straining to use your service. Making the question more open like this means you aren’t highlighting what you think are barriers but looking out for anything that could be. A caveat with these questions is that any score under 3 will open a follow up question asking about specifics.

How would you rate the ease of adoption? (Scale Question 1Low-5High)

This is in relation to how much effort it took to introduce your service into their teaching. You obviously want to pay attention to this as it is going to impact the amount of new users and the early attrition rate. Again, this has a follow up if it gets a low mark to get more useful information on what you can improve on. Removing or lowering as many barriers to adoption is going to be the biggest thing to help you grow a service.

What would you do without the Noteable service? (Free text answer)

This might be one that is more suited to our position with the Noteable service. We are offering a centralised service so it’s good to know if this is reducing a burden or making it easier for people to use this specific tool – Jupyter notebooks. As a pilot service we need to be able to say there is a justifiable need for providing this service, that without this service users would be inconvenienced.

What is the most important feature that you would like to see? (Free text answer)

Nothing like crowd-sourcing your new features….

To what extent do you agree with the following statement: “Using Noteable helped me to deliver my teaching/training” (Scale Question 1Low-5High)

Directly asking if introducing this tool has any discernible affect on the ability of our colleagues to actually deliver their teaching. It’s often very difficult to quantify where something has a pedagogic impact but by asking this directly we are getting the answer from the people who are most likely to know.

To what extent do you agree with the following statement: “Using Noteable helped my students to interact with the material” (Scale Question 1Low-5High)

Again, this is something that is difficult to quantify but can have a massive impact. If your students can more easily engage with the material it means you can get further into more complex subject, spend more time in theory and less time getting over barriers. This will be followed up with student questionnaires at a later date.

It’s important to not lose sight of how valuable this information can be, whilst you will want to shout about good feedback make sure you spend the time fixing what users have reported and working on what they have said they want in the future. Few things can improve your relationship with your users more than “you said, we did”.

This questionnaire is currently live, I will share the feedback when it is finally complete. Please let me know if you have any improvements or insights into this, I’m always trying to make this better.

p.s. if you are going to steal these questions, at least replace Noteable with [your service name]. (I have actually seen this before and it is amazing/terrible)


More about Noteable

I’ve recently tried to become a little more involved in the wider Jupyter community and I have to say that it’s the most amazing community to be part of and work with. We at the University of Edinburgh have already managed to get funding for our nbgrader workshop which has been a great excuse to introduce the University, myself and the Noteable service.

If you want to get involved, or even just be in on the discussions then first head over to Jupyter Discourse, secondly make sure to get in on the newly revived Jupyter Community meetings. The community is always full of interesting nuggets or information and with such a rapidly growing community there is always something new coming up. If you worry about not being able to keep up then I can highly advise the Tracking Jupyter Newsletter by Tony Hirst

Being involved in the community has already started to pay off with the following discovery, originally found on Discourse –

This is a great intro to what’s out there is the Jupyter ecosystem, a breakdown comparison of 6 services that allow you to run/share/preview Jupyter notebooks……but Oh Dear, Noteable isn’t on the list! Time to get our name up there with the big companies like Microsoft and Google. So here is my contribution to how Noteable compares, including some bonus points! Noteable is a service designed for use in educational institutions, currently looking for partners in the UK.

Supported languages: Python 2.7, Python 3.5, Julia, R, Sage, Matlab, Calysto, Octave, Haskell (also able to install other languages by request!)
Ability to install packages: Yes, for the duration of your session. Also we can install packages by request to avoid having to install them yourself (also we have a lot already –
Interface similarity: Uses Native Jupyter notebook interface
Keyboard shortcuts: Uses all of the same keyboard shortcuts as Jupyter
Missing features: None
Added features: nbgrader (currently in Beta)built using Docker containers to allow you to switch between environments quickly by launching a new container.
Ease of working with datasets: You can upload a dataset to your project from your local computer, and it can be accessed by any notebook.
Internet access: Yes
Ability to work privately: Yes
Ability to share publicly: Not directly but you can use Git to share your repos or send notebooks manually.
Ability to collaborate: Not currently
Ability to upgrade for better performance: As a service designed for education there is no ‘free’ version but we do offer trial access to Noteable to institutions that are interested.
Documentation and technical support: General documentation for integrating with your LMS via LTI is available. General introduction documentation to using Jupyter is currently being created.
Conclusion: Noteable is the only service developed and designed for use in education, primarily further and higher education. We have already developed a working LTI connection to fit in with your LMS and work closely with our users to add features that are of the most benefit.

If what you see above sounds like something you would be interested in then feel free to get in touch and ask for a trial for your institution.


After following the original thread on Discourse and the Google Group there have been some extra questions raised:

– Jupyter Authenticator Support: Can it work with my organizations’ existing single sign on app? e g. 2FA, TOTP, the new Webauthn, GitHub, GitLab

Noteable can work with your institutions login, with a working LTI we can also directly connect with your LMS meaning that your users can be seamlessly logged in and also directed to specific course (which becomes important with our new Assignments)

– Mobile interface: Can any of the existing iOS and Android front ends connect to the system?

Directly from Noteable we do not offer our own App but I can say that we work on Juno, this is a paid App but does allow you to try some basic notebooks for free.  They have also recently announced that they will be donating back a portion of their profits to Jupyter so a win all around.

– Backups: How does the system backup work? How frequently, to where, and are there backups tested to make sure that they work?

Noteable auto-saves your work at regular intervals (every few minutes if you are making changes). You can also choose to save a Checkpoint if you think you might need to revert.