Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CloudWatchLogs.filter_log_events nextToken is not null but there are no results #4258

Closed
agoyburu opened this issue Sep 5, 2024 · 2 comments
Assignees
Labels
bug This issue is a confirmed bug. closed-for-staleness cloudwatchlogs p2 This is a standard priority issue response-requested Waiting on additional information or feedback.

Comments

@agoyburu
Copy link

agoyburu commented Sep 5, 2024

Describe the bug

When calling filter_log_events using CloudWatchLogs client, the nextToken key is always coming with a token instead of null even though there are no results.

Expected Behavior

nextToken should be null when events is empty.

Current Behavior

events returns an empty list but nextToken is not null

Reproduction Steps

  1. Create a CloudWatchLogs client
  2. Call filter_log_events with logGroupName, filterPattern, startTime, endTime
  3. Iterate over pages with the nextToken.

Possible Solution

I don't have a possible solution but a workaround could be to also check the results if that is empty, just ignore the nextToken

Additional Information/Context

The odd thing is that this is only happening with one AWS account so far.

To do the filtering, we first call describe_log_groups to get the list of log groups and then perform the filtering for each group.

Here are the some of the arguments I used:

startTime = 1724946563020
endTime = 1725551363020
filterPattern = '{$.readOnly = false && ($.eventName = "CreateAccessKey" || $.eventName = "DeleteAccessKey" || $.eventName = "UpdateAccessKey" || $.eventName = "CreateLoginProfile" || $.eventName = "UpdateLoginProfile" || $.eventName = "DeleteLoginProfile" || $.eventName = "CreatePolicy" || $.eventName = "DeletePolicy" || $.eventName = "AttachUserPolicy" || $.eventName = "DetachUserPolicy" || $.eventName = "AttachGroupPolicy" || $.eventName = "DetachGroupPolicy" || $.eventName = "AttachRolePolicy" || $.eventName = "DetachRolePolicy" || $.eventName = "PutUserPolicy" || $.eventName = "DeleteUserPolicy" || $.eventName = "PutGroupPolicy" || $.eventName = "DeleteGroupPolicy" || $.eventName = "PutRolePolicy" || $.eventName = "DeleteRolePolicy")}'
logGroupName = 'aws-cloudtrail-logs-us'

And here some actual responses from the filter_log_events

First call

{
    'events': [],
    'searchedLogStreams': [],
    'nextToken': 'Bxkq6kVGFtq2y_MoigeqscPOdhXVbhiVtLoAmXb5jCo-J4ub4atWFi6pePqNs7mRKkYCSSs9IimWr8Xi7GBPMaUH88oHH_P8Wu9fW7nHndM7aRAIAgrkh0Lp-O9Y4FX3e2GN7q4cuKB97nMXpl5EpWbIsPOF5kwHGwIH8PTizyX4h0Gay8fTxQ2Xmd4HcPw7foC7QRm4npaaSjuyyQWnsBzb0KmS8F3RE_e8TZIpfaoF5B4jzeU0utoThPxTLaqS81MqmKYtqQLOWA-QGD1xNw',
    'ResponseMetadata': {
        'RequestId': '29daab62-8be4-4d02-997f-435d7c21cdff',
        'HTTPStatusCode': 200,
        'HTTPHeaders': {...},
        'RetryAttempts': 0,
    },
}

Second call

{
    'events': [],
    'searchedLogStreams': [],
    'nextToken': 'Bxkq6kVGFtq2y_MoigeqscPOdhXVbhiVtLoAmXb5jCojwBgZEf6LNx1087xbbrYSh9yQVDcRp0Q1KWq2KS92-zCrMPMoJ5OQezrUspC_T42pPEsvHmxc3TwOAdIuacjnipZWfZNseUTWEBd5fL4tOt3WSDov9XUT3_iscd23BFEutBi88IzZG09WVGfoYPTKWgQ1EsZyKpdqm5se-w26iUE5OeeHXTmlwY9hmVKwgd7aMEyvUQEGPufQOHAyhaNjxDebszWs5h28UyfmAuKi-Q',
    'ResponseMetadata': {
        'RequestId': 'f24a9577-4bc8-4303-9b65-dc2d3ed57ab5',
        'HTTPStatusCode': 200,
        'HTTPHeaders': {...},
        'RetryAttempts': 0,
    },
}

SDK version used

boto3==1.34.80

Environment details (OS name and version, etc.)

MacOS Sonoma 14.6.1 (23G93), Ubuntu 22.04 docker image

@agoyburu agoyburu added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Sep 5, 2024
@tim-finnigan tim-finnigan self-assigned this Sep 10, 2024
@tim-finnigan
Copy link
Contributor

Thanks for reaching out. The filter_log_events command makes a call to the underlying FilterLogEvents API, so this issue is really about the API behavior. If you add boto3.set_stream_logger('') to your script, then you can see the API request/response.

The reason you're getting a nextToken but not events is likely because the API is returning a max limit of results and there are no events matching your filter in that response. As an alternative, I recommend using the paginator to automatically paginate through responses: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs/paginator/FilterLogEvents.html

(For more info on paginators please refer to this documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html)

@tim-finnigan tim-finnigan added response-requested Waiting on additional information or feedback. cloudwatchlogs p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Sep 10, 2024
Copy link

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a confirmed bug. closed-for-staleness cloudwatchlogs p2 This is a standard priority issue response-requested Waiting on additional information or feedback.
Projects
None yet
Development

No branches or pull requests

2 participants