Sriram Krishnan: The right tool for the right job

The right tool for the right job

The problem? Delete a few repeating patterns from a few text files. Except that each text file is several hundred megabytes in size. For some weird reason, I spent *quite* some time fiddling around with sed (yes - *that* sed )and trying to get multi-line patterns to work.

And then it hit me.

Five minutes and an equal number in lines of Python code later, I had something that did the job as well.

There's a moral in this tale somewhere. I'm just not sure what it is.

# Saturday, October 21, 2006 3:50 PM

Comments:

Just wondering.. Do you use MSYS or Cygwin?

# posted by

Casanova : 11:55 PM, October 22, 2006

Cygwin - though I do have a few native ports lying around my Windows machine somewhere.

I really should try SFU (Microsoft Services For Unix)

# posted by

Sriram : 1:20 AM, October 23, 2006

A file several hundred megabytes in size? How about you'd tell us how long it took to process it :-)

# posted by

Anonymous : 6:37 AM, October 27, 2006

Less than a minute. I have nice hardware :-)

# posted by

Sriram : 3:01 PM, October 27, 2006

Do you load the entire file into memory before working on it?

# posted by

monkeeboi : 11:24 AM, November 02, 2006

Nope- I stream through

# posted by

Sriram : 11:58 AM, November 02, 2006

I take it you don't use regular expressions then.

Just curious, how do you go about looking for the pattern.

# posted by

monkeeboi : 7:39 AM, November 03, 2006

Since it is line demarcated stuff, I just need to do a .find() to see whether a particular piece of text is there in a line or not

# posted by

Sriram : 12:41 AM, November 05, 2006

Hi sriram, this is guru(HP,chennai!)...even i was working on an text file (report generated by sales team) & its need to be formatted into excel, but fixed demilit import needs manual time spent, so they asked me for a macro, i used .net vb (contains() ) code (but they where clever enough to tell me that only "--" & " " should be deleted...and i was feeling very happy to inform this to my manager that only .net help me to solve quick issues within minutes!

# posted by

Anonymous : 12:09 PM, November 06, 2006

Vista is toooo GOOD!..and i am waiting for Retail version!...did you know any idea (time frame) when it will be available in India?
(vista ultimate!) - Guru,chennai

And wish you a (delayed) Deepavali wishes!

# posted by

Anonymous : 12:10 PM, November 06, 2006

It's a mess to hear that the video's song was censurated

# posted by

Anonymous : 4:42 AM, November 12, 2006

Sriram Krishnan > Blog

The right tool for the right job

Archives