To find probability or confidence score of model output,

[Aj-232425]

Hello @jasonrig @Stallon-niranjan ,
I was working on retraining of addressnet. I did it successfully, now i want to find the confidence score / probability of model. Like how much my model is confidence (86% confident of address result generated)
For which I tried using tf.nn.softmax but it's throwing an error.
Value error:- "Truth value of an array with more than one element is ambiguous. Use a.any or a.all".

Is there any way if you guys can help me out to find out confidence score, probability function which helps me out to use addressnet over millions of addresses.

Any help would be appreciated.

Thanks & Regards
Aj.

[jasonrig]

Hey, sorry for not being so active here.
Calculating the probability of a given address is kind of difficult.
Remember that this model (assuming you're using the same architecture as I originally implemented) performs the following:

1. For each character it assigns a class probability of belonging to each address component
2. Goes through each character one-by-one, grouping them into substrings belonging to each class
3. Outputs a dictionary giving the final output4.

So actually we have a confidence per character of the input, and not of the output as a whole.

You could come up with some quality metric by taking the product of the highest class probability for each character. This would tell you the probability of all characters together being the correct class (probability multiplication rule). But the result might look a little misleading, because the product of probabilities will get smaller and smaller with the increasing length of the string. It's not nice to have a number that shrinks proportionally to the length of the address.

You could also take the average confidence of the predicted class for each letter. This would tell you that, on average, each letter was predicted with a confidence of x. It will not diminish with the string length, so might be more comparable between predictions.

[Aj-232425]

Thank you so much @jasonrig for actively replying.
Yes, i did understood what you are telling. But when i applied tf.nn.softmax to find out the probability, i am getting an error i referred in question.
Is there any way if you could please take out few minutes and do some rough coding. That would be really helpful. that would be really appreciated.
Well, i have already retrained it for US address.
I am just stuck here in confidence score, cause it's too necessary to find score or probability else I will end up checking lacs of addresses again.(cross verifying)
Thanks & Regards,
Aj

[jasonrig]

The softmax is already applied here:

address-net/addressnet/model.py

Line 31 in 28e7c2d

'probabilities': tf.nn.softmax(logits)

So you should be able to access the values here:

address-net/addressnet/predict.py

Line 143 in 28e7c2d

for char, class_id in zip(addr.upper(), res['class_ids']):

[jasonrig]

Just be aware that the value you calculate can't be interpreted as the confidence that the address is correct.
It will be related to the probability that the character labels are correct, but the address itself could still be wrong. The only real way to test the accuracy is to have some real-world, annotated, testing dataset.

[Aj-232425]

Thank you so much @jasonrig for detailed explanation, before I got chance to see your last reply, i had already applied and followed the same you explained. And yeah, you are right . What i am getting is probability of each character. Doesn't look like confidence score. Initially i was getting array of character falling in class component, found out value have maximum value from array matrix.
For example we have word "street" probability looks like "0.11,0.11,0.11,0.11" , as you said we can't consider this as confidence as it's probability of character. Neither we can go with average or product .

Of course, i have already tested on some real word actual data and using which I tried finding out accuracy. Where accuracy=precision=recall.

My aim is just to confirm, for ex, out of 1000 sample real world addresses, X% have high accuracy/confidence score/probability above some threshold of being true predicted. So that I don't need to check every sample address if it's predicted correctly or not.
If there's any suggestion, appreciated.

Thanks