Repository navigation
Keep Bernoulli training loss and prior KL in logit space - #3305
Open
AHMETHAKANBEZIR1 wants to merge 1 commit into
Open
AHMETHAKANBEZIR1 wants to merge 1 commit into
AHMETHAKANBEZIR1 wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Confident, incorrect Bernoulli predictions can stop learning because
sigmoidrounds the network output to a probability before the loss converts it back to logits. For float32 logits [-40, 40] and targets [1, 0], the current loss is capped near 15.94 and both gradients are zero; the Bernoulli NLL should be 40 with nonzero gradients. A linear model initialized with the wrong positive logit also fails to update.Related to #621. This addresses the Bernoulli case only, not every likelihood discussed in that issue.
Override
BernoulliLikelihood.compute_lossto retain the network logits for the NLL. Compute the optional KL(prior || prediction) as logit cross-entropy minus prior entropy, so it also stays finite when the predicted probability rounds to zero or one. Sample weighting still applies to the NLL, not the prior term. Sampling, probability parameter outputs, and the existing parameter-extraction methods are unchanged.Validation on Windows CPU, Python 3.12.14, PyTorch 2.14.1+cpu:
AI assistance: implementation and tests were prepared and executed autonomously with OpenAI Codex. No independent human review or personal production incident is claimed.