Storing files/images/binary data into the database involves lots of performance issues. The more widely recommend solution for such a scenario is to push the files onto disk and store the metadata about the file and its location into a database. This has its pros and cons. If you have such a need and dont have the time to build a custom solution you can try MogileFS/Compete filesystem/Bit Mountain. If you are from perl background you can use MogileFS. For python you can try CFS/BM.

MogileFs usage flow is as follows :

• app requests to open a file (does RPC via library to a tracker, finding whichever one is up). does a “create_open” request.
• tracker makes some load balancing decisions about where it could go, and gives app a few possible locations
• app writes to one of the locations (if it fails writing to one midway, it can retry and write elsewhere).
• app (client) tells tracker where it wrote to in the “create_close” API.
• tracker then links that name into the domain’s namespace (via the database)
• tracker, in the background, starts replicating that file around until it’s in compliance with that file class’s replication policy
• later, app issues a “get_paths” request for that domain+key (key == “filename”), and tracker replies (after consulting database/memcache/etc), all the URLs that the file is available at, weighted based on I/O utilization at each location.
• app then tries the URLs in order. (although the tracker’s continually monitoring all hosts/devices, so won’t return dead stuff, and by default will double-check the existence of the 1st item in the returned list, unless you ask it not to…)

In the above flow, one quirky thing is once the file is pushed on to the disk then the app tells tracker where it stored the file. Since storing of the file and metadata info is not atomic, there is high possibility of syncing issues. Will explore further to see if they have some internal mechanism to handle this quirk.

Synergy, strategic fit, core competencies, best practice, bottom line, revisit, expeditious, to tell you the truth (or the truth is), 24/7, out of the loop, benchmark, value-added, proactive, win-win, think outside the box, fast track, result-driven, knowledge base, at the end of the day, touch base, mindset, client focus(ed), paradigm, game plan, leverage.

After a very long time , i am creating an entry. Have started to attend webinars on Search Technologies and am planning to blog my learnings. Probably it will be kind of notes to the webinars.

To get an high level overview of search architecture, read the Anatomy of Large-Scale hypertext Web Search Engine by Brin and Page , which they published during their stint in Standford.

Search involves 3 major processes.

1. The Spider / Crawler that crawls all the web pages.

Care should be taken that the crawlers we write should adhere to standards like providing identification about from where the crawler is coming from, adhereing to the robots.txt file which indicates which part6s of the site the crawler can crawl and which crawlers can crawl etc and also should ensure that we dont bring down sites by running our crawlers.

2. The Indexing process

Inverted indexes are used to store the crawled data. The indexing process involves generating list of keywords from the content of the page , the proximity of the key words , location of the words etc are generated and stored in the index with uniquely generated document id.

3. The Lookup or Retrieval Process with Ranking.

The search servers provide us with the UI to enter the search terms we want to search on. Once the terms are submitted, the search servers look up the inverted index to obtain the documents that have search terms as keywords. Once it has the list it ranks the documents based on the frequency of the apprearence of the search terms, the location of the terms, the anchor text pointing to the document, the importance of sites that points to that particular document etc. The documents get displayed based on ranking they get through the ranking process. One famous algorithm is Google’s Page Rank algorithm.

Some interesting links to learn about search engines :

www.searchenginewatch.com
www.searchenginejournal.com
www.searchengineshowdown.com

http://battellemedia.com

www.searchtools.com

Happy Learning !!

Utility to convert BIN/CUE CD image to ISO

bchunk xyz.bin xyz.cue xyz

Command to mount iso images in linux

mount -o loop -t iso9660 file.iso /mnt/test