English Word Frequency Lists

Many might have come across a requirement for reasonable sized English word frequency lists. Here is one good and free word frequency list based on  British National Corpus (BNC). This post is just a pointer to the real resource (Reference 1), but I will copy some text from the reference describing the details about the file structure.

-----------------------------------------------
These are all available in 6 forms:

• sorted alphabetically ("al") or by frequency (highest frequency first) ("num");
• the complete lists, or a smaller file containing only those items occurring over five times (suffix "o5");
• all lists are available compressed using gzip (".gz"). The

o5 lists are also available uncompressed (no suffix). The frequencies are for <CLAWS-word, POS> pairs.
For a list and brief descriptions of CLAWS POS-tags, see here.

The format is: four fields, separated by spaces.

 1: frequency
 2: word
 3: pos
 4: number of files the word occurs in

For non-orthographic words, spaces are replaced by underscore, giving eg "in_spite_of".
Lists are provided for the complete BNC (all), and for three subsets, as below:

 cg 'context-governed' spoken material    
  (eg meetings, lectures etc)  6.2M tokens,  79,906 types
 demog   'demographic' spoken material        
  (eg conversation)      4.2M tokens,  54,652 types
        written                             89.7M tokens, 921,074 types
 all                             100.1M tokens, 939,028 types

File sizes in MB ("al" and "num" variants all the same size) are:

  all uncompressed .gz o5 o5.gz
-------------------------------------------------------------
all  18.1   4.8 4.0 1.32
cg   1.4   0.39 0.43 0.15
demog   0.9   0.26 0.25 0.09 
written  17.8   4.7 3.9 1.30
-------------------------------------------------------------

For all.al.gz click here
For all.al.o5 click here
For all.al.o5.gz click here
For all.num.gz click here
For all.num.o5 click here
For all.num.o5.gz click here
For written.al.gz click here
For written.al.o5 click here
For written.al.o5.gz click here
For written.num.gz click here
For written.num.o5 click here
For written.num.o5.gz click here
For cg.al.gz click here
For cg.al.o5 click here
For cg.al.o5.gz click here
For cg.num.gz click here
For cg.num.o5 click here
For cg.num.o5.gz click here
For demog.al.gz click here
For demog.al.o5 click here
For demog.al.o5.gz click here
For demog.num.gz click here
For demog.num.o5 click here
For demog.num.o5.gz click here

References:
1. http://www.kilgarriff.co.uk/bnc-readme.html

Python Multiprocess programming - A Pool of Workers

Python's Global Interpreter Lock prevents multithreading to actually be parallelizable, and sometimes making it much slower than single thread version. To obtain true parallelism, we need to use multiple processes.

Here is the code to start a certain number of processes, and pass tasks to them through a queue. The done_queue.get() blocking function ensures that the parent process remains active until the child threads finish everything. Once we obtain all the results, we stop the child processes using the "STOP" input (as even the child processes also block on input.get()).

import time
import random

from multiprocessing import Process, Queue, current_process, freeze_support

#
# Function run by worker processes
#

def worker(input, output):
    for func, args in iter(input.get, 'STOP'):
        result = calculate(func, args)
        output.put(result)

#
# Function used to calculate result
#

def calculate(func, args):
    result = func(*args)
    return '%s says that %s%s = %s' % \
        (current_process().name, func.__name__, args, result)

#
# Functions referenced by tasks
#

def mul(a, b):
    time.sleep(2*random.random())
    return a * b

def plus(a, b):
    time.sleep(2*random.random())
    return a + b

#
#
#

def test():
    NUMBER_OF_PROCESSES = 8
    TASKS1 = [(mul, (i, 7)) for i in range(20)]
    TASKS2 = [(plus, (i, 8)) for i in range(10)]

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    # Submit tasks
    for task in TASKS1:
        task_queue.put(task)

    # Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

    # Get and print results
    print 'Unordered results:'
    for i in range(len(TASKS1)):
        print '\t', done_queue.get()

    # Add more tasks using `put()`
    for task in TASKS2:
        task_queue.put(task)

    # Get and print some more results
    for i in range(len(TASKS2)):
        print '\t', done_queue.get()

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')


if __name__ == '__main__':
    freeze_support()
    test()

References: 
1. https://docs.python.org/dev/library/multiprocessing.html

Microsoft Excel Bar/Column Chart Patterns

It is possible to create GNUPlot style bar charts with patterns (shown below)  in Excel.

As discussed in Reference 2, for Excel 2007 you need to download the excel macro file "Pattern Fill Add-In". The usage instructions are provided in Reference 1.

References:
1. http://www.andypope.info/charts/patternfills.htm
2. http://answers.microsoft.com/en-us/office/forum/office_2007-excel/excel-2007-bar-chart-shading/99bab79a-e600-4bbe-a08e-448363d3173d

Microsoft Excel Clustered-Stacked Column/Bar Chart

We can create clustered-stacked column/bar charts -- like those shown below (taken from Reference 1) -- in Excel. It is doable, but the process involves many steps. Follow the instructions in Reference 1.

At one point, the instructions say - "Copy the range, and use paste special to add this data to the chart as a new series", for this if using Excel 2007 you need to click the 'Home' tab on the ribbon, click the 'Paste' drop-down arrow and choose 'Paste Special'.

References:
1. http://peltiertech.com/WordPress/clustered-stacked-column-bar-charts/
2. http://www.mrexcel.com/forum/excel-questions/559998-charts-add-new-data-copy-paste-special.html

Changing Windows Internet time synchronization frequency

You might come across situations when you need multiple Windows machines to have their times synchronized. You could setup time synchronization protocols between the machines, making one as the master and the others as slaves.

An easier alternative is to sync all the machine times to Internet times - the NIST time or Windows server time available by default in the Date & Time settings. Once you sync the machines with the Internet time, the machines might start drifting away over a period. Hence Windows periodically syncs the times with the Internet time; however, the default sync duration ranges from few days to weeks.

You can make the time sync happen much more frequently - like few hours. To do this, you need to make system registry changes. The following instructions are for Windows XP machine; for other versions, the process might be similar.

1. Go to Start-> Run and type "regedit" to open the registry settings.

2. Navigate to the following in the Registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient

3. The parameter you need to change is SpecialPollInterval
Right click on 'SpecialPollInterval' in the right hand column and select Modify>Decimal. The decimal number is in seconds. For example, you can set it to 86400 to mean 24 hours.

NOTE: Please backup the Registry beforehand to avoid any issues later (I do not know how this can be done).

Reference:
1. http://www.tomshardware.com/forum/55562-45-changing-frequency-internet-time-synchronisation 

English words present in dictionary for Python use

A good resource of English words available in English dictionary is WordNet by Princeton University. Though its primary usage is not for getting the list of English words and their meanings, you can still get that information from WordNet.

Either you could download the WordNet database and do the parsing yourself (or using other pre-existing Python scripts), or use the NLTK natural language processing toolkit in Python. The toolkit provides instructions for using wordnet at WordNet Interface.

You could also use the PyEnchant spell checking library available for Python. Using this you can check whether a word is acceptable as per a specific English dictionary, though you won't get the word meanings.

As stated in [2], the word coverage in WordNet is not complete, but it is very useful. Depending on the use case, one option can be preferred over the other.

References:
1. http://www.velvetcache.org/2010/03/01/looking-up-words-in-a-dictionary-using-python
2. http://stackoverflow.com/questions/3788870/how-to-check-if-a-word-is-an-english-word-with-python
3. http://www.nltk.org/howto/wordnet.html
4. http://pythonhosted.org/pyenchant/

Delete "Not Existing" files in Windows 7

Sometimes even after we delete a file or move it, its ghost copy might still remain in the original folder. The ghost file might have been created due to a race condition. When we try to delete the file, we get an error stating "Cannot Delete the File, The file does not exist".

In that case, use a command prompt in windows and navigate to the folder. Use "dir /x" command to see if the ghost file is being listed. As per the reference [1], if you can identify the 8.3 format filename in the output, you can use it to delete the file using "del <filename>" command from the prompt. An easier way would be to use wildcard filename to delete the ghost file. The wildcard command might look like " del *file.txt".


References:
1. http://superuser.com/questions/388860/cant-delete-files-that-do-not-exist-but-appear-in-my-download-directory