pandas check if all rows of a dataframe are equal or same

I could not easily find solutions for this problem hence this post.

Here are two ways of checking that
df.apply(pd.Series.nunique, axis=0).unique().tolist() == [1]
and
df.groupby(df.columns.tolist()).ngroups == 1

the groupby method was *slightly* faster in my timing tests using the following code

import pandas as pd
import timeit

data = [
dict(zip(list('abc'), range(1, 4))),
dict(zip(list('abc'), range(1, 4))),
]

df = pd.DataFrame(data)

print(df.apply(pd.Series.nunique, axis=0).unique().tolist() == [1])

print(df.groupby(df.columns.tolist()).ngroups == 1)

print(timeit.timeit('df.apply(pd.Series.nunique, axis=0).unique().tolist() == [1]',
number=1000, globals=globals()))

print(timeit.timeit('df.groupby(df.columns.tolist()).ngroups == 1', number=1000,
globals=globals()))


Advertisements

letsencrypt fullchain.pem = cert.pem + chain.pem

TL;DR fullchain.pem is concatenation of cert.pem and chain.pem

I get the TLS certificates for nginx web server via letsencrypt. They provide the following 4 files on successful authentication

• cert.pem
• chain.pem
• fullchain.pem
• privkey.pem

here is how you verify that fullchain.pem is just a concatenation of cert.pem and chain.pem. Please note that the order of those two files is important.

diff fullchain.pem <(cat cert.pem chain.pem)

pip install –user spacy 2

this is NOT about installing spacy and models to a virtual environment. i faced no problems in doing that.

this is NOT about installing spacy and models system wide using sudo pip install.

i want to use pip install --user to install spacy and models to ~/.local/lib/python3.x/site-packages

first: install spacy using pip install --user -U spacy OR python -m pip install --user -U spacy.

second: find the shortcut name of the model you want to install. i want to install the model en_core_web_lg https://spacy.io/models/#available-models

third: issue the following command in a terminal and as soon as you see the URL to the model hit Control-c to cancel the progress. the command is python -m spacy download en_core_web_lg.
the reason we don’t want the command to complete is that it will try to install the model to the system wide site-packages directory
the other reason why i do it this way is that spacy automatically tells me the name of the compatible model. i don’t want to download a model that is incompatible with the installed version of spacy.
in my case the URL was https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz

fourth: download that model using curl or wget or some other program.
curl -L -O https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz

fifth: pip install –user /path/to/downloaded/model pip install --user -U en_core_web_lg-2.0.0.tar.gz

sixth: now link the model with some shortcut name. this name is to be used later to load the model. python -m spacy link en_core_web_lg en. here en is the shortcut name i chose while en_core_web_lg is the name of the installed model.

finally: verify the whole thing using python -m spacy validate. you should be able to load the model using

import spacy
nlp = spacy.load('en') # shortcut name


multiple colons in makefiles

more than one colon in gnu make makefile rule

gnu make makefiles more than one colon

i could not find the following page while googling for the search queries mentioned above. hope google is able to send people to this page at which point they will find what they are looking for

https://www.gnu.org/software/make/manual/html_node/Static-Usage.html#Static-Usage

default user agent in scrapy

scrapy settings --get=USER_AGENT

Scrapy/1.2.1 (+http://scrapy.org)

scikit-learn NLP list english stopwords

from sklearn.feature_extraction import stop_words

print(stop_words.ENGLISH_STOP_WORDS)

currently there are 318 words in that frozenset.

NLTK also has its own stopwords

from nltk.corpus import stopwords
print(stopwords.words('english'))

there are 153 words in that

apache hive divide a column by its sum

select a/sum(a) from foo; -- WRONG!

will not work because sum(a) would work on each row and will return the number present in that row resulting in all 1. since sum(n) = n where n is a single number.

here is where we can make good use of  hive windowing and analytics functions.

the solution is to use the following:

select a/sum(a) over () from foo; -- RIGHT !!!

here we are instructing that the sum be performed over the entire column ‘a’.

you could of course make 2 queries and calculate the sum of a first and then hardcode the sum in another query but that means you are performing 2 passes over the table for sure. now i can’t be sure how many passes over the table are being made in the right query above. if you know the answer please post in the comments below.