JLeRN node upgraded

The node was previously running version 0.23.5 of the LR code but is now running version 0.23.7.  Version 0.23.6 was a major release and 0.23.7 was a minor release.

The release notes for version 0.23.6 state the following:

• Data Services NEW
  • The extract API is included and enabled in this distribution. The Extract API brings a “batteries included” interface for tailoring data extraction and query to suit more narrow and varied use case needs without high resource requirements.
  • Data Service views are NOT installed by default. Node administrators can install any or all of the 3 standards alignment focused data services using python CouchApp. This is done to prevent unneeded consumption of disk space for data services not utilized.
  • Interactive documentation explaining Data Services and how to roll your own for your use case.
• Resource Data Distribution refactoring CHANGED
  • Distribute has been significantly refactored to reduce the need for 2x storage for documents. New distribute uses 2 DB’s, incoming and resourcedata. New _tainted documents can be distributed to incoming, and node may impose an internal policy for untainting documents before moving into resource_data. All harvest services still operate against resource_data.
  • More compatible with Learning Registry 0.23 specification. Document Types of resource_data are distributed, not resource_data_distributable.
  • Upgrade will require some minor configuration changes to NGINX to change or expose incoming endpoint.
  • IMPORTANT Legacy nodes can distribute to a 0.23.6 by adjusting service documents to use the incoming endpoint. Legacy cannot be the destination for a 0.23.6 server.
• Subscription for Distribute NEW
  • You may now visit http://%5Bnode address]/register, enter the URL of your node (use https if SSL required to access).
• Support for CouchDB 1.2.0 NEW
  • This is a highly recommended upgrade. Significantly improves the storage and resource utilization, as well paves path for features planned for future releases.
• Many Bug fixes

I have not yet installed the Data Services views (pertaining to standards alignment) but if anybody wants them please tell me.

Next steps:
1. Test the new Extract API
2. Test distributing data between nodes

I’ll report back with the test results.

The Windows Node at Mimas

Worked to establish a node on the Windows Server 2008 machine and after some tweaking and self learning got through quite far installing the node.There were some issues on getting Nginx setup right though.

Then after some tweaking, Nginx goes smoothly on the Windows 2008 server after restarting the machine, but wasn’t sure why the step to push the couchapps not working. I was trying to navigate to the config folder of the LR repository
cd C:\Python27\LearningRegRepository\config
and ran the command

python setup_node.py

and was getting the following error:

Traceback (most recent call last):
File “setup_node.py”, line 12, in
import couchdb
ImportError: No module named couchdb

Checked the python code, and couchdb folder is very much existing.. well I knew now is the time to “DEBUG” which I guess I love the most. Lou on the US Learning Registry team suggested to download version 1.1.1 from https://github.com/dch/couchdb/downloads

Damon Regan from the US Learning Registry team noticed the blog post and was happy about the work on Windows machine. He mentioned that Lou Wolford on the LR team has worked with getting the LR running on Windows. There is an active pickling error that they’re working to resolve. He shared the discussion thread on that: https://github.com/LearningRegistry/LearningRegistry/pull/167 and discussions were happening on the Windows mods during the dev calls. He hoped that all issues get resolved and I can get it up and running soon.

Later on I managed to resolve as per Jim’s suggestion for Ubuntu on https://groups.google.com/forum/#!msg/learningreg-dev/0sKsLb15fi8/hFFhObPk69IJ

”
cd ~/gitrepos/LearningRegistry/LR
pip install -e ./

That should install ALL LR dependencies outside of wsgi, then:

cd ~/gitrepos/config
python setup_node.py -d
”
I used the same version of couchDB as Lou suggested. The pickling error was still persisting and I was going through the discussion thread Damon mentioned in his comment.

Lou commented that they are looking into a solution that looks for what OS is running and switches between threads/processes to keep everything stable no matter what OS is being used. He promised to keep me updated and let me know ASAP when it is released.  Damon mentioned that John Poyau from the LR team is working on a new factory model to allow windows platforms to use threads and linux and mac platforms to use process. Then in few days, John was able to fix the issue by switching to use Thread instead of Process in change_monitor to fix the pickling_error in Windows. Also he tweaked the way test files are declared to be Windows-friendly.

Damon mentioned that the fix is tested on Windows, Mac and linux. He also mentioned that the fix is not formally released yet. but is in their stable master. He recommended to update our experimental windows node at Mimas from the stable master to obtain the fix. He suggested following steps as well to upgrade the node to master

”
1. Pull the most recent tag from git

cd /LearningRegistry
git checkout master
git pull

2. Run the setup node python script

python setup_node.py -d
”

Then he mentioned that it is important to re-run the setup node python script as configuration changes take place during updates. The LR team hopes to have an update script soon that will preserve our node settings on update.

Now I have followed Damon’s steps and do not see the pickling error anymore, and able to start the server on localhost. Watch this space for further development on the Windows environment.

Alpha node

We installed the JLeRN “alpha” node on our new Ubuntu server on the weekend (Jan 21-22) before the Hackday.  But there was a bug which affected the indexes and thus document retrieval.

The LR developers in the U.S. resolved the bug promptly and we applied the fix on Jan 28th.

The node is here: http://alpha.mimas.ac.uk

The node supports all publish and retrieval services although we haven’t had a chance to test all of these yet.

The node is now open to the world except you need a username and password to publish documents.  The credentials you need to publish are:

Username: fred
Password: flintstone

Give it a go when you have a minute or two.

Examples:
http://alpha.mimas.ac.uk/status
http://alpha.mimas.ac.uk/description
http://alpha.mimas.ac.uk/obtain  [all documents in node, 100 at a time]
http://alpha.mimas.ac.uk/obtain?request_ID=http://dspace.jorum.ac.uk/xmlui/handle/123456789/1000
http://alpha.mimas.ac.uk/slice?any_tags=curve
http://alpha.mimas.ac.uk/slice?any_tags=magnetism

N.B.: The examples above fetch JSON documents.  Install an extension to your favourite browser so you can display the documents in a readable format; e.g., JSONview extension for Firefox.

Curl examples:
$ curl  -v http://alpha.mimas.ac.uk/obtain?request_ID=”http://alpha.mimas.ac.uk/obtain?request_ID=http://dspace.jorum.ac.uk/xmlui/handle/123456789/1000″
$ curl -Xv POST -H “Content-Type:application/json” “http://alpha.mimas.ac.uk/publish” -d @test_data.json -u fred

The Learning Registry Quick Reference Guide has more examples.

Node of Mimas

Nick here.  I work at Mimas with Sarah and Bharti.  I installed a node in mid-December on a spare machine I had lying around.  As it was nothing “official” I called it Node of Mimas.  One simple way of looking at the node’s status is to execute a command like this:

$ curl http://<…>/status

{ “node_name”: “Node of Mimas”, “node_id”: “7ee34ddd8e4f4ea4aafd59c3c7619a16”, “active”: true, “timestamp”: “2012-01-11T20:32:01.097068Z”, “start_time”: “2011-12-31T10:01:07.698981Z”, “install_time”: “2011-12-14T14:44:24.206372Z”, “earliestDatestamp”: “2011-12-14T15:34:23”, “doc_count”: 1 }

You might notice the output from this command is a pretty JSON document.  This node was installed on December 14th and last restarted on New Year’s Eve.  It contains one document.  I wonder what it is?  Execute the following command to find out:

$curl http://<…>/obtain

{“documents”:[{“document”: [{“doc_type”: “resource_data”, “resource_locator”: “URI_of_resource”, “resource_data”: “Put_anything_like_metadata, xml_or_whatever_here”, “update_timestamp”: “2011-12-14T15:34:23.219869Z”, “keys”: [“science”, “what_ever_you_want”], “TOS”: {“submission_TOS”: “http://www.learningregistry.org/tos/cc0/v0-5/&#8221;}, “_rev”: “1-f81b1258b28092a314661527ad7dcbf0”, “resource_data_type”: “metadata”, “payload_placement”: “inline”, “payload_schema”: [“hashtags”, “describing”, “resource”, “format”], “node_timestamp”: “2011-12-14T15:34:23.219869Z”, “doc_version”: “0.23.0”, “create_timestamp”: “2011-12-14T15:34:23.219869Z”, “active”: true, “publishing_node”: “7ee34ddd8e4f4ea4aafd59c3c7619a16”, “_id”: “551592f1743d46a7b4f4d6c7484e356a”, “doc_ID”: “551592f1743d46a7b4f4d6c7484e356a”, “identity”: {“owner”: “”, “submitter”: “Your name or organization here”, “submitter_type”: “agent”, “signer”: “Your name or organization if signing the document”, “curator”: “”}}], “doc_ID”: “URI_of_resource”}]}

Let’s make the output more readable for humans (and edit it slightly):

Sample document

Well, it’s just a test document, in JSON format, but it gives you an indication of what Learning Registry data looks like.

The Hackday in Manchester is fast approaching.  We are expecting to be given a new server by the end of this week but it will take a few days to install and test a new node we are calling Alpha.  By the middle of next week we will decide which node to use for the Hackday, either Node of Mimas or Alpha.  That’s all for now.