+1 vote

I am trying to implement the cluster estimation method using EM found in Weka, more precisely the following description:

The cross validation performed to determine the number of clusters is done in the following steps:

1. the number of clusters is set to 1
2. the training set is split randomly into 10 folds.
3. EM is performed 10 times using the 10 folds the usual CV way.
4. the loglikelihood is averaged over all 10 results.
5. if loglikelihood has increased the number of clusters is increased by 1 and the program continues at step 2.

My current implementation is as follows:

```def estimate_n_clusters(X):
"Find the best number of clusters through maximization of the log-likelihood from EM."
last_log_likelihood = None
kf = KFold(n_splits=10, shuffle=True)
components = range(50)[1:]
for n_components in components:
gm = GaussianMixture(n_components=n_components)

log_likelihood_list = []
for train, test in kf.split(X):
gm.fit(X[train, :])
if not gm.converged_:
raise Warning("GM not converged")
log_likelihood = np.log(-gm.score_samples(X[test, :]))

log_likelihood_list += log_likelihood.tolist()

avg_log_likelihood = np.average(log_likelihood_list)

if last_log_likelihood is None:
last_log_likelihood = avg_log_likelihood
elif avg_log_likelihood+10E-6 <= last_log_likelihood:
return n_components
last_log_likelihood = avg_log_likelihood```

I am getting a similar number of clusters both trough Weka and my function, however, using the number of clusters n_clusters estimated by the function

```gm = GaussianMixture(n_components=n_clusters).fit(X)
print(np.log(-gm.score(X)))```

Results in NaN, since the -gm.score(X) yields negative results (about -2500). While Weka reports Log likelihood: 347.16447.

My guess is that the likelihood mentioned in step 4 of Weka is not the same as the one mentioned in the functionscore_samples().

Can anyone tell where I am getting something wrong?

Thanks Sep 26, 2018 in Python 504 views

## 1 answer to this question.

For future reference, the fixed function looks like:

```def estimate_n_clusters(X):
"Find the best number of clusters through maximization of the log-likelihood from EM."
last_log_likelihood = None
kf = KFold(n_splits=10, shuffle=True)
components = range(50)[1:]
for n_components in components:
gm = GaussianMixture(n_components=n_components)

log_likelihood_list = []
for train, test in kf.split(X):
gm.fit(X[train, :])
if not gm.converged_:
raise Warning("GM not converged")
log_likelihood = -gm.score_samples(X[test, :])

log_likelihood_list += log_likelihood.tolist()

avg_log_likelihood = np.average(log_likelihood_list)
print(avg_log_likelihood)

if last_log_likelihood is None:
last_log_likelihood = avg_log_likelihood
elif avg_log_likelihood+10E-6 <= last_log_likelihood:
return n_components-1
last_log_likelihood = avg_log_likelihood``` answered Sep 26, 2018 by
• 58,080 points

## How to calculate square root of a number in python?

calculate square root in python >>> import math ...READ MORE

–1 vote

## How to filter out any digit that contains odd number in a range of number in Python?

n = list(range(10)) b = list(filter(lambda i:i%2!=0,n)) print(b) READ MORE

## How to count the number of elements in a list?

To count the number of elements of ...READ MORE

## How to find exponent of a number in Python?

Yes you can do it using the ...READ MORE

## how do i change string to a list?

suppose you have a string with a ...READ MORE

## how can i randomly select items from a list?

You can also use the random library's ...READ MORE

+1 vote

## how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

–1 vote