[IPython-User] SAGE notebook vs iPython notebook

http://mail.scipy.org/pipermail/ipython-user/2012-January/008981.html

Hi,

What is relation and comparison between iPython notebook  and SAGE

notebook? Can someone provide motivation and roadmap for iPython notebook

as alternative to SAGE notebook? Will sometime iPython notebook support

Cython in same easy way as SAGE notebook?

P.S. I am newbie and prefer to stay with my custom set of Python and

iPython ecosystem, still curious.

Thanks,

Oleg

They're distinct things, although I think SAGE is looking into building thenext version of their notebook to use the same protocol.Why are we recreating a notebook, when SAGE's already works? I expect otherpeople will weigh in with different motivations, but first off, it workswithout a 400MB download ;-). SAGE is kind of an ecosystem of its own,bundling a lot of libraries into its package system. That has variousconsequences:- Linux distributions won't package SAGE, whereas soon you'll be able to do"apt-get install ipython notebook"- To run SAGE on Windows, you set up a virtual machine to run Linux.IPython can run natively.- SAGE ships its own copy of Python (by the looks of it, that's currentlyversion 2.6.4). IPython can be installed on other versions, includingPython 3.None of the above is criticising SAGE - the project has its own goals, andwhat they're doing makes sense for that. But for IPython, it's important tofit into the general Python landscape. Although we are in touch with SAGE,and hopefully there's room to collaborate and share code.I don't think inline Cython support is on the roadmap at present - itsounds like the sort of thing that would make for a good extension. Ifyou're interested in writing one, have a look here:http://ipython.org/ipython-doc/stable/config/extensions/index.html

Another reason: Sage is GPL and IPython is BSD.  It doesn't matter for

most users, but for those for whom it does, it's a real plus.

With the caveat Aaron mentioned, that sage being GPL, we must ask them

for explicit permission if we want to reuse any of their code (and

when we've done so, they've graciously agreed to relicense snippets as

BSD).  In contrast, sage can freely use code from ipython, which it

does.

Actually I have a very concrete plan on how to enable sage-style

'cell-level magics'.  The only problem is that the entire draft is in

my brain only and I haven't had the time to sit down and type it out.

The priority for this is obviously going up, so I'll do my best to

flush it soon.

Hi All,

I have noticed that the tab completion in certain cases produces quite

unexpected results. If I open a notebook without anything, and try to

load pylab via the magic machinery, then I get the following:

In [] : %pyl + TAB -> %pylab

In []: pyl + TAB -> pyllab

In []: pyla + TAB -> pylaab

Similarly,

In []: mag + TAB -> maggic

In []: %mag + TAB -> %magic

So, when I drop the % sign, the command is expanded incorrectly. I don't

see this behaviour in the qt frontend, because in that case the command

is not completed at all, only the possible completions are shown.

I was wondering whether it was supposed to be like this (I don't think

so), and whether I should file a bug report.

Hi,On Thu, Jan 5, 2012 at 8:06 PM, Oleg Mikulchenklo <olmikul@gmail.com> wrote:>What is relation and comparison between iPython notebook  and SAGE notebook?>Can someone provide motivation and roadmap for iPython notebook as>alternative to SAGE notebook? Let me try to provide some perspective on this, since it's a validquestion that is probably in the minds of others as well.  This isjust *my* take on it, and other devs are welcome to pitch in as wellwith their view.  Apology in advance, this is quite long, but I'mtrying to do justice to many years of development, multipleinteractions between the two projects and the contributions of manypeople.  I apologize in advance to anyone I've forgotten (but pleasedo correct me as I want to have a full record that's reasonablytrustworthy).Let's go back to the beginning: when I started IPython in late 2001, Iwas a graduate student in physics and had used extensively firstMaple, then Mathematica, both of which have notebook environments.  Ialso used Pascal (earlier) then C/C++, but those two (plus IDL fornumerics) were the interactive environments that I knew well, and myexperience with them shaped my view.  In particular, I was a heavyuser of the Mathematica notebooks and liked them a lot.I started using Python in 2001 and liked it, but the interactiveprompt felt like a crippled toy compared to the systems above or to aUnix shell.  When I found out about sys.displayhook, I realized thatby putting in a callable object, I would be able to hold state andcapture previous results for reuse.  I then wrote a python startupfile to provide these features, giving me a 'mini-mathematica' inpython by also loading Numeric and Gnuplot.  Thus was my'ipython-0.0.1' born, 259 lines to be loaded as $PYTYHONSTARTUP.  Incase you are curious, I'm attaching it here, it's kind of funny that Ican 'release' IPython 0.0.1 as an email attachment...I also read an article(http://onjava.com/pub/a/python/2001/10/11/pythonnews.html) thatmentioned two good interactive systems for Python, LazyPython and IPP. I contacted their authors,  Nathan Gray and Janko Hauser, seeking tojoin forces to create IPython together.  They were both very graciousand let me use their code, but didn't have the time to participate inthe effort.  As any self-respecting graduate student with adissertation deadline looming would do, I threw myself full-time intobuilding the first 'real' IPython by merging my code with both oftheirs.  Eventually I did graduate, by the way.The point of this little trip down memory lane is to indicate thatfrom day 1, Mathematica and its notebooks (and the Maple worksheetsbefore) were in my mind as my 'ideal' environment for dailycomputational scientific work. In 2005 we had two Google SoC studentsand we took a stab at building, using WX, a notebook system.  RobertKern then put some more work into the problem, but unfortunately thatprototype never really became fully usable.In early 2006, William Stein organized what was probably the firstSage Days at UCSD and invited me; William and I had been in touchsince 2005 as he was using IPython for the sage terminal interface.  Isuggested Robert come as well, and he demoed the notebook prototype hehad at that point.  It was very clear that the system wasn'tproduction ready, and William was already starting to think about anotebook-like system for sage as well. Eventually he started workingon a browser-based system, and by Sage Days 2 in October 2006, asshown by the coding sprint topics(http://wiki.sagemath.org/sd2-sprint), the sage notebook was alreadyusable.Sage going at it separately was completely reasonable and justified:we were moving slowly and by that point even we weren't convinced thewx approach would go anywhere. William is a force of nature and wastrying to get sage very usable very fast, so building somethingintegrated for his needs was certainly the right choice.We continued working on ipython, and actually had another attempt at anotebook-type system in 2007. By that point Brian and Min had come onboard and we had built the Twisted-based parallel tools. Using this,Min got a notebook prototype working using an SQL/SQLAlchemy backend.Like Sage this used a browser for the client but retained the 'IPythonexperience', something the Sage notebook didn't provide.This is a key difference of our approach and the Sage nb, so it' worthclarifying what I mean: the Sage notebook took the route of using thefilesystem for notebook operations, so you can't meaningfully use 'ls'in it or move around the filesystem yourself with 'cd', because sagewill always execute your code in hidden directories with each cellactually being a separate subdirectory.  This is a perfectly validapproach and lets the notebook do many useful things, but it is alsovery different from the ipython model where we always keep the uservery close to the filesystem and OS.  For us, it's really importantthat you can access local scripts, use %run, see arbitrary filesconveniently, as in data analysis and numerical simulation we makeextensive use of the filesystem.  So the sage model wasn't really agood fit for us.Furthermore, we wanted a notebook that would provide the entire'IPython experience', meaning that magics, aliases, syntax extensionsand all other special IPython features worked the same in the notebookand terminal.  The sage nb reimplemented some of these things in itsown way: they reused the % syntax but it has a different meaning, theytook some of the ipython introspection code and built their own x?/??system, etc. In some cases it's almost like ipython, in others thebehavior is fairly different, which is fine for Sage but doesn't workfor us.So we continued with our own efforts, even though by then the Sagenotebook was fairly mature by this time.  For a number of reasons (Ihonestly don't recall all the details), Min's browser-based notebookprototype also never reached production quality.Eventually, in 2009 we were able to fund Brian to dig into the heartof the beast, and attack the fundamental problem that made ipythondevelopment so slow and hard: the fact that the main codebase was anoutgrowth of that original merge from 2001 of my hack, IPP andLazyPython, by now having become an incomprehensible and terriblyinterconnected code with barely any test suite.  Brian was able todevote a summer full-time to dismantling these pieces and reassemblingthem so that they would continue to work as before (with only minimalregressions), but now in a vastly more approachable and cleanlymodularized codebase.This is where early 2010 found us, and then zerendipity struck: whileon a month-long teaching trip to Colombia I read an article aboutZeroMQ (http://lwn.net/Articles/370307) and talked to Brian about it,as it seemed to provide the right abstractions for us with a simplermodel than Twisted.  Brian then blew me away, by writing in just twodays a new set of clean Cython-based bindings: we now had pyzmq!  Itbecame clear that we had the right tools to build a two-processimplementation of IPython that could give us the 'real ipython' butcommunicating with a different frontend, and this is precisely what wewanted for cleaner parallel computing, multiprocess clients and anotebook.  When I returned from Colombia I had a free weekend anddrove down to his place, and in just two days we had a prototype of apython shell over zmq working, proving that we could indeed buildeverything we needed.Shortly thereafter, we had discussions with Enthought who offered tosupport Brian and I to work in collaboration with Evan Patterson, andbuild the Qt console using this architecture.  Our little prototypehad been just a proof of concept, but this support allowed us to spendthe time necessary to apply the same ideas to the real IPython. Brianand I would build a zeromq kernel with all the IPython functionality,while Evan built a Qt console that would drive it using ourcommunications protocol.  This worked extremely well, and by late 2010we had a more or less complete Qt console working.In October 2010 James Gao (a Berkeley neuroscience graduate student)wrote up a quick prototype of a web notebook, demonstrating that thekernel design really worked well and could be easily used by acompletely different client.  And then in the summer of 2011, Briantook James' prototype and built up a fully working system, this timeusing the Tornado web server (which ironically, we'd looked at inearly 2010 as a candidate for our communications, but dismissed it asit wasn't really the tool for that job), JQuery, CodeMirror andMathJax.  That's the notebook that we then polished over the next fewmonths to finally release in 0.12.As this long story shows, while it's taken us a very long time to gethere, what we have now makes a lot of sense for us, even consideringthe existence of the Sage notebook and how good it is for many usecases.Our notebook is just one particular aspect of a much larger and richerarchitecture built around the concept of a Python interpreterabstracted over a JSON-based, explicitly defined communicationsprotocol (http://ipython.org/ipython-doc/rel-0.12/development/messaging.html). Even considering http clients, the notebook is still just onepossible client: you can easily build an interface that only evaluatesa single cell with a tiny bit of javascript like the Sage single cellserver, for example.Furthermore, since Min also reimplemented the parallel machinerycompletely with pyzmq, now we have one truly common codebase for allof IPython. We still need to finish up a bit of integration betweenthe interactive kernels and the parallel ones, but we plan to finishthat soon.We deliberately wrote the notebook to be a lightweight, single-userprogram meant to keep its files next to the rest of your scripts andother files.  The sage notebook draws many parallels with the googledocs model, requiring a login and showing all of your notebookstogether, kept in a location separate from the rest of your files.  Incontrast, we want the notebook to just start like any other programand for the ipynb files to be part of your normal workflow, ready tobe version-controlled just like any other script or file and easy tomanage on their own.There are other deliberate differences of interface and workflow:- We keep our In/Out prompts explicit because we have an entire systemof caching variables that uses those numbers, and because thosenumbers give the user a visual clue of the execution order of cells,which may differ from the document's order.- We deliberately chose a structured JSON format for our documents.It's clear enough for human reading while allowing easy and powerfulmachine manipulation without having to write our own parsing.  Sowriting utilities like a rst or latex converters (as we recentlyshowed) is very easy.- Our move to zmq allowed us (thanks to Thomas' tireless work) to shipthe notebook working both on python2 and python3 out of the box.  Thesage notebook only works on python2, and given their use of Twisted itwill be probably some time before they can port to python3.- Because our notebook works in the normal filesystem, and lets youcreate .py files right next to the .ipynb just by passing --script atstartup, you can reuse your notebooks like normal scripts, import fromthem, etc.  I'm not sure how to import a sage notebook from a normalpython file, or if it's even possible.- We have a long list of plans for the document format: multi-sheetcapabilities, latex-style preamble, per-cell metadata, structuralcells to allow outline-level navigation and manipulation such as inLyX, ... For that, we need to control the document format ourselves sowe can evolve it according to our needs and ideas.As you see, there are indeed a number of key differences between ournotebook and the sage one, but there are very good technical reasonsfor this (in addition to the licensing point already mentioned).  Thenotebook integrates with our architecture and leverages it; you canfor example use the interactive debugger via a console or qtconsoleagainst a notebook kernel, something not possible with the sagenotebook.I'd like to close by emphasizing that we have had multiple, productivecollaborations with William and other Sage devs in the past, and Iexpect that to continue to be the case.  On certain points thatcollaboration has already led to convergence; e.g. the new Sage singlecell server uses the IPython messaging protocol, after we workedclosely with Jason Grout during a Sage Days event in March 2011 thanksto William's invitation.  In the future we may find other areas wherewe can reuse tools or approaches.  It is clear to us that the Sagenotebook is a fantastic system, it just wasn't the right fit forIPython; I hope this very long post illustrates why.Whew, that was a lot!  I probably should turn this into a blog post atsome point... Don't hesitate to ask questions on this, I promise muchshorter replies in the future :)Cheers,