Some excerpts:

• Data-driven predictions are responsible for a massive transformation of America’s dairy cows. While other industries are just catching on to this whole “big data” thing, the animal sciences — and dairy breeding in particular — have been using large amounts of data since long before VanRaden was calculating the outsized genetic impact of the most sought-after bulls with a pencil and paper in the 1980s.
• The bull market (heh) can be reduced to one key statistic, lifetime net merit, though there are many nuances that the single number cannot capture. Net merit denotes the likely additive value of a bull’s genetics. The number is actually denominated in dollars because it is an estimate of how much a bull’s genetic material will likely improve the revenue from a given cow
• Thousands of years of qualitative breeding on family-run farms begat cows producing a few thousand pounds of milk in their lifetimes; a mere 70 years of quantitative breeding optimized to suit corporate imperatives quadrupled what all previous civilization had accomplished

Interestingly, they were able to predict the perfect bull with the best quality of semen (with the highest lifetime net merit) before the bull produced even a single offspring!

Thanks to Tabby for pointing this article out!

Read the full article here on Economic Times

NimbleTV will truly revolutionize television watching!

As per the NY Times review :

NimbleTV is the latest example of technology companies trying to break into the closed system of television distribution in the United States. As Americans buy more smartphones and tablet computers, cable and satellite distributors are under pressure to provide TV access on more screens, and entrepreneurs are — depending on one’s view — either helping to provide it, or forcing it to happen on their own terms.

Large companies like Apple and Google have taken steps to enter the television distribution business.. So have start-ups like NimbleTV, which is backed by the venture capital firms Greycroft Partners and Tribeca Venture Partners and by the Tribune Company, the owner of 23 TV stations.

“We’ve all heard about TV Everywhere for a long time. One of the questions that’s bothered me is, why is it not here yet?” said Anand Subramanian, the chief executive of NimbleTV, during a preview of the service at Greycroft’s Midtown Manhattan office last week.

Way to go NimbleTV, and thanks for allowing us to participate in this exciting adventure!

The core feature of the Connector is to provide the ability to read MongoDB data into Hadoop MapReduce jobs, as well as writing the results of MapReduce jobs out to MongoDB. Users may choose to use MongoDB reads and writes together or separately, as best fits each use case. Our goal is to continue to build support for the components in the Hadoop ecosystem which our users find useful, based on feedback and requests.

For this initial release, we have also provided support for:

• writing to MongoDB from Pig (thanks to Russell Jurney for all of his patches and improvements to this feature)
• writing to MongoDB from the Flume distributed logging system
• using Python to MapReduce to and from MongoDB via Hadoop Streaming.

Though it is quite early in it’s evolution, an exciting possibility this introduces is to write MapReduce scripts for MongoDB using Hadoop Streaming. This was surely an important part and as they explain, the toughest part of the connector:

Hadoop Streaming was one of the toughest features for the 10gen team to build. To that end, look for a more technical post on the MongoDB blog in the next week or two detailing the issues we encountered and how to utilize this feature effectively.

We are really looking forward to using the connector. The ability to use MongoDB as a document store and write MapReduce scripts to process them sounds like a promising capability!

Here is a quick excerpt of the key points:

• Big Data’s manifesto: don’t be afraid
  For the first time in the technology world, we’re thinking about how to collect more data and analyze it, instead of how to reduce data and archive what’s left
• A Big Data Definition
  Big Data is the area of tech concerned with procurement and analysis of very granular, event-driven data
• 2012 – Year of Big Data?
  Big Data is moving from specialized use in science and tech companies to Enterprise IT applications. The quest to make Big Data more Enterprise-friendly should result in the refinement of the technology and lowering the costs of operating it
• Spreadmarts aren’t Big Data, but they have a role
  Big Data lets older, conventional technologies provide insights on data sets that cover a much wider scope of operations and interactions than they could before
• Natural language processing and Big Data
  Big Data technology itself will help to improve natural language technology as it will allow greater volumes of written works to be processed and algorithmically understood
• Big Data specialists and developers: can they all get along?
  People who develop strong competency in both – building software as well as procurement and analysis of data that software produces and consumes – will be rare, and thus in demand and very well-compensated.
• The feds and Big Data?
  The recent $200 million investment in Big Data announced by the U.S. Federal government is an indication of the importance of harnessing Big Data in the work of the government and its importance to society
• Big Data and BI are separate, but connected
  Big Data is its own subcategory and will likely remain there. But it’s part of the same food chain as BI and data warehousing.

And a nice conclusion:

We’re taking huge amounts of data, much of it unstructured, using cheap servers and disks. And then we’re on-boarding that sifted data into our traditional systems. We’re answering new, bigger questions, and a lot of them. We’re using data we once threw away, because storage was too expensive, processing too slow and, going further back, broadband was too scarce. Now we’re working with that data, in familiar ways — with little re-tooling or disruption. This is empowering and unprecedented, but at the same time, it feels intuitive.

That’s revolutionary.

Read the entire article here

Enterprises are well equipped to measure and manage the core transactional part of the business using structured data stored in conventional databases. However, managing the core transactional business is now merely a necessity and not sufficient to be competitive in today’s market. With every business transaction today there is as an increasing volume of unstructured data such as documents, emails, sms messages, instant messages, log files (such as web access logs, email logs etc.). Enterprises are feeling the need to store, manage and analyze this unstructured data.

However, conventional enterprise applications that excel at managing structured data are not really capable of handling the daunting size of such unstructured data, or even if they do, the cost of such infrastructure required can easily overtake the value added by managing such data.

Fortunately, there are new technologies that have evolved to help manage a combination of structured and unstructured data without requiring enterprises to squirm at the costs.

If you consider the two primary determinants of costs involved in managing data:

1. Storage Capacity
2. Processing Capacity

Conventional technologies required you to plan the required storage and processing capacity in advance, so that you can provide for the worst case scenario of the maximum possible storage and peak processing capacity required through the lifetime of the data. It was not easy to dynamically scale up the costs proportional to actual capacity utilization.

This is where the new technologies like Cloud Computing and Big Data differ.

Cloud computing has made it easy for enterprises to dynamically scale up or down their infrastructure in small incremental steps, based on actual required storage and processing capacity.

Big Data technologies complement this by providing data management software that can manage large volume of data at high speeds, by using a cluster of multiple low-capacity servers. It does this by distributing the storage and processing of the data across the entire cluster efficiently.

In addition to being well suited for leveraging low-cost cloud infrastructure, most big data technology is available in the open source. This means, license costs are minimal. However, professional level support and maintenance services are available. Reach1to1 provides development, support, implementation and training on a wide range of big data technologies.

Users today are spoiled by the speed and ease of use of publicly available Internet applications. They can search for anything and get instant results from search engines like Google, connect with their friends across the world using social applications like Facebook, find professionals for any skill or expertise as well as jobs on networking applications like LinkedIn, and share their life experiences in real-time using applications like Twitter.

Contrasting with these fast and easy to use applications are conventional enterprise applications that now seem too restrictive, slow and cumbersome. These new applications are based on new technologies that enable such fast and reliable performance. And the best part is this – in spite of the fact that there have been several man years of effort put into them, most of these new technologies are available as open source.

This provides a huge opportunity for enterprises to leverage open source platforms such as Hadoop and related technologies to create reliable and highly scalable data repositories using a distributed set of low cost commodity servers.

Hadoop originated in 2006 as a project of the Apache Software Foundation. The primary components it is composed of are  MapReduce and the a Hadoop Distributed File System (HDFS).

Hadoop is capable of processing large amounts of data – running into petabytes – at very high speeds. It is able to scale up the volume of data without significantly affecting the performance due to the fact that both, the storage and processing of data is not centralized in a large database server, but distributed in a scalable network of low cost commodity servers.

— editing new content above this line —

Realiability

As businesses become aware that the Big Data trend is here to stay, publishers are looking for reliable support. There are many forms or reliability, all of which will have an effect on the overall reliability of the instrument and therefore the data collected. Reliability is an essential pre-requisite for validity. It is possible to have a reliable measure that is not valid, however a valid measure must also be reliable.

Scalability

scalability is the ability of a system, network, or process, to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.[1] For example, it can refer to the capability of a system to increase total throughput under an increased load when resources (typically hardware) are added.

Distributed Computing

– Multiple architectures and use cases
– Focus today: using multiple servers, each working on part
of of job, each doing same task job, each doing same task
– Key Challenges:

• Work distribution and orchestration
• Error recovery
• Scalability and management

• scanning for trigger events and sending only interesting data to the network
• improving data quality by performing real-time averaging or digital filtering
• numerically intensive data pre-processing avoid overloading processors of receiving machines.

• Assessing mixed data (structured and unstructured) from multiple sources
• Dealing with unpredictable content with no apparent schema or structure
• Enabling real-time or near-real-time collection, analysis, and answers

Big Data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high-velocity capture, discovery, and/or analysis.

Big data spans three dimensions: Volume, Velocity and Variety.

• Volume – Big data comes in one size: large. Enterprises are awash with data, easily amassing terabytes and even petabytes of information.
• Velocity – Often time-sensitive, big data must be used as it is streaming in to the enterprise in order to maximize its value to the business.
• Variety – Big data extends beyond structured data, including unstructured data of all varieties: text, audio, video, click streams, log files and more.

Big data is more than a challenge; it is an opportunity to find insight in new and emerging types of data, to make your business more agile, and to answer questions that, in the past, were beyond reach. Until now, there was no practical way to harvest this opportunity.