Skip to content

Tool Usage Counters Counting Behavior #319

@wrightsh

Description

@wrightsh

Hello,

I believe there's a bug with how the tool usage counters count. My use case is I have a tool that uses gold and I want to track how much gold we've deposited so that we know when to refill. I've setup a pre and post Tool Usage Question that's a number field named "gold", and I also have "tool_control_prefill_post_usage_with_pre_usage_answers" enabled. It doesn't quite work as I would expect, when I test it I get:

  • Tool Usage Counter on Pre + (pre & post data match) --> counts both times (double counting for this use case)

  • Tool Usage Counter on Pre + (pre & post data don't match) --> counts on start but not end (with this setup the pre is someone's plan, but the post is what actually happened, maybe the tool broke and they were unable to do the run)

  • Tool Usage Counter on Post + (pre & post data match) --> doesn't count either

  • Tool Usage Counter on Post + (pre & post data don't match) --> counts ending value

I was testing out changing _update_tool_counters to be something like this:

 def _update_tool_counters(self, usage_event: UsageEvent, run_data_json: Dict):
        # This function increments/decrements all counters associated with the given tool
        if usage_event.pre_run_data and not usage_event.run_data:
            pre_post = "pre"
        elif usage_event.run_data:
            pre_post = "post"
        else:
            return

        counter_question_name = f"tool_{pre_post}_usage_question"

        active_counters = ToolUsageCounter.objects.filter(is_active=True, tool_id=usage_event.tool_id)
        active_counters = active_counters.filter(**{f"{counter_question_name}__isnull": False})

        for counter in active_counters:
            additional_value = 0
            counter_question_field = getattr(counter, counter_question_name)
            for question in self.questions:
                input_data = run_data_json[question.name] if question.name in run_data_json else None
                additional_value += get_counter_value_for_question(question, input_data, counter_question_field)
                if isinstance(question, PostUsageGroupQuestion):
                    for sub_question in question.sub_questions:
                        additional_value += get_counter_value_for_question(
                            sub_question, input_data, counter_question_field
                        )
            if additional_value:
                counter.value += counter.counter_direction * additional_value
                counter.save()

With the same example I get:

  • Tool Usage Counter on Pre + (pre & post data match) --> counts only the pre

  • Tool Usage Counter on Pre + (pre & post data don't match) --> counts only the pre

  • Tool Usage Counter on Post + (pre & post data match) --> counts only the post

  • Tool Usage Counter on Post + (pre & post data don't match) --> counts only the post

And for this use case I think I would just use the counter on Post.

I've also noticed that the number fields minimum doesn't seem to work if you set it to 0.

Thanks!
Shawn

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions